Quantized Stochastic Gradient Descent: Communication versus Convergence

Parallel implementations of stochastic gradient descent (SGD) have received signif1 icant research attention, thanks to excellent scalability properties of this algorithm, 2 and to its efficiency in the context of training deep neural networks. A fundamental 3 barrier for parallelizing large-scale SGD is the fact that the cost of communicat4 ing the gradient updates between nodes can be very large. Consequently, lossy 5 compression heuristics have been proposed, by which nodes only communicate 6 quantized gradients. Although effective in practice, these heuristics do not always 7 provably converge, and it is not clear whether they are optimal. In this paper, we 8 propose Quantized SGD (QSGD), a family of compression schemes which allow 9 the compression of gradient updates at each node, while guaranteeing convergence 10 under standard assumptions. QSGD allows the user to trade off compression and 11 convergence time: it can communicate a sublinear number of bits per iteration 12 in the model dimension, and can achieve asymptotically optimal communication 13 cost. We complement our theoretical results with empirical data, showing that 14 QSGD can significantly reduce communication cost, while being competitive with 15 standard uncompressed techniques on a variety of real tasks. 16